[BugFix] Unschedule global RF timer in PipelineDriver destructor by jenwitteng · Pull Request #76252 · StarRocks/starrocks

jenwitteng · 2026-07-13T04:15:50Z

Why I'm doing:

A BE/CN aborts during shutdown with std::bad_weak_ptr:

terminate called after throwing an instance of 'std::bad_weak_ptr'
  what():  bad_weak_ptr
  ... starrocks::pipeline::RunTimerTask(void*)
  ... bthread::TimerThread::Task::run_and_delete()
  ... bthread::TimerThread::run()

The global runtime-filter timer (RFScanWaitTimeout) is owned solely by
PipelineDriver::_global_rf_timer, and it is only unscheduled in
finalize(). Queued or blocked drivers that are abandoned when the driver
executor is closed at shutdown are destroyed without going through
finalize(), so the bthread TimerThread — which keeps only a raw pointer to
the task — can still fire on a task whose last owning shared_ptr is already
gone. PipelineTimerTask::doRun() then calls shared_from_this() on the freed
object and throws std::bad_weak_ptr, aborting the process.

This is the abandoned-driver counterpart to the query-context-destruction race
fixed in #73082, which unscheduled only from finalize(). The throwing
shared_from_this() in doRun() was introduced when PipelineTimerTask
became enable_shared_from_this in #72058.

What I'm doing:

Extract the unschedule into _unschedule_global_rf_timer() and call it from
both finalize() and ~PipelineDriver().
The helper moves the shared_ptr into a local so the task stays alive across
unschedule_and_join(), closing the window where a concurrently running
doRun() could observe a zero use_count.
Add unit tests: the destructor unschedules a pending global RF timer, and the
destructor blocks until an in-flight timer task returns.

Verified: both changed translation units compile cleanly with the project
toolchain (gcc-12, -std=gnu++23), zero diagnostics.

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
- This pr needs auto generate documentation
This is a backport pr

Bugfix cherry-pick branch check:

I have checked the version labels which the pr will be auto-backported to the target branch
- 4.1
- 4.0
- 3.5

CelerData-Reviewer · 2026-07-13T04:16:37Z

@codex review

A queued or blocked PipelineDriver that is abandoned when the driver executor is closed during shutdown is destroyed without going through finalize(), which was the only place the global runtime-filter timer (RFScanWaitTimeout) was unscheduled. Because the bthread TimerThread keeps only a raw pointer to the task, it can then fire on a task whose sole owning shared_ptr (PipelineDriver::_global_rf_timer) is already gone. PipelineTimerTask::doRun() calls shared_from_this() on that freed object and throws std::bad_weak_ptr, aborting the process: terminate called after throwing an instance of 'std::bad_weak_ptr' RunTimerTask -> bthread::TimerThread::run This is the abandoned-driver counterpart to the query-context-destruction race fixed in StarRocks#73082, which only unscheduled from finalize(); the throwing shared_from_this() in doRun() was introduced in StarRocks#72058. Move the unschedule into a helper called from both finalize() and ~PipelineDriver(). The helper moves the shared_ptr into a local so the task stays alive across unschedule_and_join(), closing the window where a concurrently running doRun() could observe a zero use_count. Add unit tests covering the destructor unscheduling a pending timer and the destructor blocking until an in-flight timer task finishes. Signed-off-by: Jenwit Amonpongitsara <jenwit.amonpongitsara@agoda.com>

chatgpt-codex-connector · 2026-07-13T04:20:23Z

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: b94e4bb620

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

stdpain

It's OK for a hot fix

github-actions · 2026-07-13T05:33:08Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2026-07-13T05:33:11Z

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2026-07-13T05:54:59Z

[BE Incremental Coverage Report]

✅ pass : 7 / 7 (100.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/exec/runtime/pipeline_driver.cpp	7	7	100.00%	[]

github-actions · 2026-07-13T09:01:09Z

@Mergifyio backport branch-4.1

github-actions · 2026-07-13T09:01:09Z

@Mergifyio backport branch-3.5

github-actions · 2026-07-13T09:01:11Z

@Mergifyio backport branch-4.0

mergify · 2026-07-13T09:01:14Z

backport branch-4.1

✅ Backports have been created

Details

#76277 [BugFix] Unschedule global RF timer in PipelineDriver destructor (backport #76252) has been created for branch branch-4.1 but encountered conflicts

Cherry-pick of b0363d9 has failed:

On branch mergify/bp/branch-4.1/pr-76252
Your branch is up to date with 'origin/branch-4.1'.

You are currently cherry-picking commit b0363d9b0e.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   be/src/exec/pipeline/pipeline_driver.h

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   be/src/exec/pipeline/pipeline_driver.cpp
	both modified:   be/test/exec/pipeline/schedule/pipeline_timer_test.cpp

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

mergify · 2026-07-13T09:01:15Z

backport branch-3.5

✅ Backports have been created

Details

#76278 [BugFix] Unschedule global RF timer in PipelineDriver destructor (backport #76252) has been created for branch branch-3.5 but encountered conflicts

Cherry-pick of b0363d9 has failed:

On branch mergify/bp/branch-3.5/pr-76252
Your branch is up to date with 'origin/branch-3.5'.

You are currently cherry-picking commit b0363d9b0e.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   be/src/exec/pipeline/pipeline_driver.h

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   be/src/exec/pipeline/pipeline_driver.cpp
	both modified:   be/test/exec/pipeline/schedule/pipeline_timer_test.cpp

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

mergify · 2026-07-13T09:01:24Z

backport branch-4.0

✅ Backports have been created

Details

#76279 [BugFix] Unschedule global RF timer in PipelineDriver destructor (backport #76252) has been created for branch branch-4.0 but encountered conflicts

Cherry-pick of b0363d9 has failed:

On branch mergify/bp/branch-4.0/pr-76252
Your branch is up to date with 'origin/branch-4.0'.

You are currently cherry-picking commit b0363d9b0e.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   be/src/exec/pipeline/pipeline_driver.h

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   be/src/exec/pipeline/pipeline_driver.cpp
	both modified:   be/test/exec/pipeline/schedule/pipeline_timer_test.cpp

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

github-actions Bot added 4.0 3.5 4.1 labels Jul 13, 2026

mergify Bot assigned jenwitteng Jul 13, 2026

github-actions Bot requested review from ZiheLiu and stdpain July 13, 2026 04:17

jenwitteng force-pushed the fix/main/pipeline-driver-dtor-unschedule-global-rf-timer branch from b94e4bb to 2c184a5 Compare July 13, 2026 04:18

stdpain approved these changes Jul 13, 2026

View reviewed changes

trueeyu approved these changes Jul 13, 2026

View reviewed changes

trueeyu merged commit b0363d9 into StarRocks:main Jul 13, 2026
63 checks passed

github-actions Bot removed 4.1 3.5 labels Jul 13, 2026

github-actions Bot removed the 4.0 label Jul 13, 2026

Uh oh!

Conversation

jenwitteng commented Jul 13, 2026

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

Uh oh!

CelerData-Reviewer commented Jul 13, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 13, 2026

Uh oh!

stdpain left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 13, 2026

[Java-Extensions Incremental Coverage Report]

Uh oh!

github-actions Bot commented Jul 13, 2026

[FE Incremental Coverage Report]

Uh oh!

github-actions Bot commented Jul 13, 2026

[BE Incremental Coverage Report]

file detail

Uh oh!

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

mergify Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

mergify Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

mergify Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mergify Bot commented Jul 13, 2026 •

edited

Loading

mergify Bot commented Jul 13, 2026 •

edited

Loading

mergify Bot commented Jul 13, 2026 •

edited

Loading